Fix checkpoints during uploads not being applied #558
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Under our consistency guarantee, we can't make data from new checkpoints visible if there are local changes that haven't been uploaded yet. Our current implementation for this case (recognizable by
ready = false
) was to simply do nothing. That's not unreasonable: After all, the pending local changes are supposed to be uploaded eventually, which should trigger new checkpoint messages that we'd then apply.However, there is a potential race condition when a pending upload is currently in progress. While the connector is uploading changes (and before finishing), some data may already be uploaded to the database, triggering the sync service to send new checkpoints. These checkpoints would be entirely ignored, because we're already uploading data and there is no subsequent upload to trigger another one.
To fix that issue, this PR improves the checkpoint handling logic. If we get a checkpoint complete message that didn't apply because of local data or a
write_checkpoint
mismatch, and we're currently in the process of uploading changes or waiting for awrite-checkpoint2.json
request, then await that before trying the same checkpoint again. Either,write-checkpoint2.json
returns). Or,